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FOREWORD 


Typically,  R&D  Utilization  Reports  discuss  the  effective  utilization  of  research  developed  by 
the  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  or  under  contract  to 
ARI  Final  reports  contain  a more  detailed  presentation  of  the  research  and  are  available  on  a 
limited  basis 


In  response  to  specific  requirements  of  the  Training  Management  Institute  of  the  Army 
Training  and  Doctrine  Command  (TRADOC),  Fort  Eustis,  Virginia  and  of  Army  Project 
2Q763731A770  "Performance-Oriented  Individual  Skill  Development,"  ARI  has  developed  a 
system  of  hands-on  performance  testing  through  Skill  Qualification  Tests  (SQT).  The  Fluman 
Resources  Research  Organization  (HumRRO)  developed  the  initial  draft  Performance  Test 
Development  for  Skill  Qualification  Testing'  A Manual  by  Robert  Vineberg  and  Elaine  N.  Taylor 
in  August  1975,  based  partly  on  projects  PERFORM  and  ATC  PERFORM,  and  used  this  draft  to 
conduct  workshops  on  SQT  development,  under  Contract  DAHC  19  75-C  0016  (Dr.  Theodore  R. 
Powers,  Principal  Investigator).  ARI  is  continuing  development  and  validation  of  the  SQT,  with 
rela'ed  programs  by  ARI  Field  Units  at  Fort  Ord,  California  and  Fort  Knox,  Kentucky 


IMPLEMENTING  THE  SKILL  QUALIFICATION  TESTING  SYSTEM 


INTRODUCTION 

As  part  of  the  Army's  efforts  to  upgrade  its  personnel  programs,  the 
Enlisted  Personnel  Management  System  (EPMS)  was  established  in  1974-  to 
design  a career  system  for  the  individual  enlisted  man  and  provide  the 
requisite  number  of  men  in  each  skill  level  of  every  Military  Occupa- 
tional Specialty  (MOS''  necessary  to  carry  out  the  Army's  mission.  In 
order  to  accomplish  this,  EPMS  provides  training  and  instruction  as 
part  of  a career- long  process  for  enlisted  personnel  as  well  as  a com- 
prehensive testing  program  that  will  assess  the  competence  of  soldiers 
at  all  skill  levels.  Soldier's  Manuals  contain  training  and  evaluation 
guidelines  for  individual  MOS  to  address  the  former  concern;  Skill 
Qualification  Tests  (SQT)  are  being  developed  to  address  the  latter. 

Beginning  in  January  19rr'’,  SQTs  are  scheduled  to  gradually  replace 
the  present  paper-and-pencil  MOS  tests.  SQTs  are  composed  of  performance 
tests  and  performance-based  tests  of  tasks  at  specific  skill  levels  of 
an  MOS.  The  development  and  validation  procedures  will  result  in  stand- 
ardized, job-related  tests.  These  tests  will  yield  reliable  results; 
they  must  also  be  acceptable  to  soldiers,  supervisors,  and  commanders  as 
valid  measures  of  ability;  they  must  be  suitable  for  administration 
anywhere  under  a variety  of  conditions.  The  Army  Training  and  Doctrine 
Command  (TRADOC)  sponsored  research  on  SQT  development  by  the  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARl).  The 
research  results  were  used  to  formulate  doctrine  implemented  by  the 
Test  Development  Activities. 


SPECIFICATIONS  FOR  SKILL  QUALIFICATION  TESTS 

The  goal  of  Skill  Qualification  Tests  is  accurate  identification  of 
mastery  or  nonmastery  of  critical  job  tasks.  To  do  this,  SQTs  sample 
the  critical  job  tasks  of  an  MOS  by  means  of  the  Scoreable  Unit.  Each 
Scoreable  Unit  is  designed  to  identify  whether  or  not  an  individual  can 
perform  one  critical  task  to  acceptable  standards  under  a given  set  of 
conditions.  It  is  composed  of  a set  of  Performance  Measures  for  tasks 
measured  through  hands-on  performance  tests,  or  a set  of  Items  for  tasks 
measured  through  performance  or  performance-based  written  tests.  SQTs 
are  composed  of  55~75  Scoreable  Units  (Figure  1). 

Performance  Measures  and  written  Items  are  designed  to  evaluate 
performance  of  crucial  elements  of  the  tasks.  Each  Performance  Measure 
or  Item  must  indicate  ability  on  an  element  of  the  task;  i.e.,  masters 
succeed,  while  nonmasters  fail.  Performance  Measures  or  Items  are 
grouped  into  Scoreable  Units  for  which  reasonable  standards  of  task 
mastery  can  be  established.  An  aggregate  score  of  GO  on  the  Scoreable 
Unit  defines  mastery,  while  NO-GO  defines  nonmastery.  GO/NO-GO  scores 
on  Scoreable  Units  are  summed  to  yield  a total  SQT  score  which  provides 
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Figure  1 . Diagrammatic  structure  of  the  SQT. 


estimates  of  MOS  proficiency.  The  focus  of  attention  during  the  develop- 
ment of  SQTs,  i.e.,  Scoreable  Unit  construction  and  validation,  is  on 
performance  and  performance-based  tests  of  tasks.  Accuracy  of  all 
Scoreable  Units  is  empirically  evaluated  during  the  iterative  validation 
procedure.  Validity  against  an  external  criterion  is  of  less  concern 
for  criterion  referenced  proficiency  tests  than  for  the  tradihional 
achievement  and  aptitude  tests  because  validity  is  more  inherent  in 
criterion-referenced  development  and  validation  procedures. 
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CONSTRUCTION  AND  VALIDATION  OF  SCOREABLE  UNITS 


Tables  1 and  2 summarize  the  development  of  SQT  Scoreable  Units. 


Table  1 

SUMMARY  OF  STEPS  TO  CONSTRUCT  SCOREABLE  UNITS 


1.  REVIEW  TASK  ANALYSIS 

2.  SELECT  TASKS  FOR  SQT 

3.  CATEGORIZE  TASKS 

A.  Primarily  Physical  Skills  (go  to  step 

B.  Primarily  Mental  Skills  (go  to  step  5) 

U.  PHYSICAL  SKILLS  TASKS  — DETERMINE  FEASIBILITY  FOR 

A.  Hands-on  Component 

YES  - develop  hands-on  Scoreable  Unit 

NO  - determine  feasibility  for  Performance  Certification 
Component 

B.  Performance  Certification  Component 

YES  - put  tasks  in  Performance  Certification  Component 
NO  - develop  performance-based  written  Scoreable  Unit 

C.  Performance-Based  Test  For  A Written  Scoreable  Unit 

3.  MENTAL  SKILLS  TASKS 

A.  Develop  Written  Scoreable  Units 

B.  Consider  Hands-on  And  Performance  Certification 
Components  For  Unusually  Long,  Complex  Tasks 
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Table  2 


OUTLINE  OF  SKILL  QUALIFICATION  TEST  SCOREABLE  UNITS  VALIDATION  PROCEDURES 


PHASE  I.  HANDS-ON  COMPONENT 

II.  WRITTEN  COMPONENT 

0 • Construct  Scoreable  Units,  at 

least  00  percent  extra 

• 

Construct  Scoreable  Units,  at 
least  00  percent  extra 

• Test  Scoreable  Units  with 
experts 

• 

Administer  Scoreable  Units 
to  item  writers 

. Identify  mastery 
. Determine  acceptability 

• 

Try  out  2 Scoreable  Units 
against  performance  te'ts 

Try  out  Scoreable  Units  on  5 
novices  with  4 raters 
. Compute  rater  agreement 
. Compute  proportion  pass 
Determine  acceptability  to 
5 novices  and  4 raters 


. Identify  mastery 
. Compute  Agreement  Indices 
between  items  and  performance 
test 

Try  out  Scoreable  Units  on  $0 
examinees 

. Identify  experts  and  novices 
with  self-ratings 
. Compute  Agreement  Indices 
between  self-ratings  and 
Scoreable  Units 
. Compute  item-Scoreable  "nit 
agreement 

Determine  acceptability  to 
examinees 

Conduct  formal  review  with  5 
Subject  Matter  Experts 


II  • Try  out  Hands-on  Component 

with  representative  soldiers, 
raters,  and  evaluators  * 

. Compute  rater  agreement 
. Judge  administrative  feasi- 
bility 

, Determine  acceptability  to 
10  soldiers,  4 raters,  and 
evaluators  * 

• Revise 

• Forward  Hands-on  Component 
with  agreement  and  accepta- 
bility data  to  ITEG  b 


III  • Determine  acceptability 

• Revise,  reassemble  component, 
and  forward  to  ITEG  b 


• Try  out  Scoreable  Units  on 
OO-^jO  representative  soldiers 
. Compute  item-Scoreable  Unit 

agreement 

. Determine  acceptability  to 
soldiers  and  evaluators  a 

• Revise 

• Forward  Written  Component  with 
agreement  and  acceptability 
data  to  ITEG  b 


• Revise  Scoreable  Units  as 
requested  by  ITEG  b 


* evaluators'  10  supervisors  and  5 commanders 
b ITEG:  Individual  Training  Evaluation  Group 
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PRODUCTS 


ARI  contracted  with  the  Human  Resources  Research  Organization  (llumRRO) 
to  produce  a manual  for  developing  hands-on  performance  tests.  In  August 
1 i , however,  the  requirements  tor  SQTs  were  changed  by  TRADOC  to 
include  both  hands-on  and  written  components.  Procedures  for  the  re- 
structured SQT  were  presented  by  ARI  at  the  EFMS  Implementation  Conference, 
August  1 T’lj.  The  papers  were  entitled  Skill  Qualification  Test  Design 
and  Validation  by  Proponent  Agencies  and  Procedures  for  Scoring  the  Skill 
Qualification  Test. 

The  draft  manual  prepared  under  contract  and  the  validation  procedures 
prepared  by  ARI  formed  the  basis  of  a series  of  four  workshops  presented 
by  HumRRO  for  Test  Development  Activities  during  August,  September,  and 
October  1075*  Concurrently,  HumRRO  was  preparing  another  draft  of  the 
manual  to  incorporate  written  Scoreable  Units.  The  revised  manual  was 
distributed  by  TRADOC  to  Test  Development  Activities  on  '5  February  lQ7r  . 
As  part  of  this  continuous  effort  papers  were  presented  at  the  Military 
Testing  Association  Annual  Conference  in  September  1175  by  Drs.  Hirshfeld 
and  Young  of  ARI,  CPT  H.  W.  Forthouse  of  TRADOC,  and  Drs.  Taylor,  Vineberg, 
and  Osborn  of  HumRRO. 

ARI  submitted  Procedures  for  Validating  Skill  Qualification  Tests  to 
TRADOC  in  October  I775.  The  Training  Management  Institute  of  TRADOC 
distributed  the  validation  procedures  to  all  Test  Development  Activities 
on  11  November  1 775*  Meanwhile,  ARI  had  been  participating  with  selected 
Test  Development  Activities  in  the  development  and  validation  of  their 
SQTs.  This  interaction  led  to  revision  of  the  validation  procedures;  the 
revised  validation  procedures  were  submitted  to  TRADOC,  who  distributed 
them  to  the  Test  Development  Activities  in  April  107b. 


UTILIZATION 

The  development  manual  and  validation  plan  are  being  used  in  the 
context  of  the  EFMS  objectives  to  provide  not  only  training  but  testing 
that  assesses  competence  of  soldiers  at  all  skill  levels.  They  arc  being 
used  initially  by  nine  Test  Development  Activities  (infantry,  Armor,  Air 
Defense,  Ordnance,  Engineer,  Quartermaster,  Administration,  Military 
Police,  and  Signal)  as  these  test  developers  construct  and  empirically 
validate  Scoreable  Units.  The  experience  of  this  first  group  will  lead 
to  improved  test  procedures  for  later  groups.  Subsequently  these 
procedures  will  be  used  by  the  ^0+  Test  Development  Activities. 

The  SQT  system  can  be  generalized  to  civilian  personnel  selection, 
placement,  and  promotion  actions. 
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